Skip to content

feat(llamaindex): add native LlamaIndex SDK integration#355

Merged
Abhijeet Prasad (AbhiPrasad) merged 13 commits intomainfrom
feat/llamaindex-integration
Apr 27, 2026
Merged

feat(llamaindex): add native LlamaIndex SDK integration#355
Abhijeet Prasad (AbhiPrasad) merged 13 commits intomainfrom
feat/llamaindex-integration

Conversation

@starfolkai
Copy link
Copy Markdown
Contributor

@starfolkai starfolkai Bot commented Apr 27, 2026

Summary

  • Adds native LlamaIndex instrumentation via BraintrustSpanHandler and BraintrustEventHandler registered on LlamaIndex's root dispatcher
  • Captures all LlamaIndex operations: LLM calls, query engines, retrievers, synthesizers, embeddings, agents, document processing
  • Wired into auto_instrument() — works with a single call
  • Extracts data beyond OTEL baseline: model config, retrieval scores, node metadata, synthesis context, embedding dimensions, reranking details

Files

  • py/src/braintrust/integrations/llamaindex/ — new integration directory (8 files)
  • py/src/braintrust/auto.py — added llamaindex parameter
  • py/src/braintrust/integrations/__init__.py — registered integration
  • py/noxfile.py — added test_llamaindex nox session
  • py/pyproject.toml — added version matrix, dep group, cassette-dirs

Test plan

  • test_integration_setup — handler registration verified
  • test_integration_idempotent — no duplicate handlers
  • test_auto_instrument_includes_llamaindex — registry entry works
  • test_document_processing — SentenceSplitter spans captured
  • test_span_hierarchy — parent-child relationships preserved
  • test_llm_error_handling — errors captured on spans
  • VCR tests need cassette recording: nox -s "test_llamaindex(latest)" -- --vcr-record=all

Closes #350

🤖 Generated with Claude Code

@starfolkai starfolkai Bot force-pushed the feat/llamaindex-integration branch from 240b786 to 5900bd5 Compare April 27, 2026 19:37
Nova (SFK) and others added 12 commits April 27, 2026 18:23
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove event handler (redundant with span handler data extraction),
remove separate helpers module, flatten classify/extract logic, drop
per-span integration metadata noise, fix VCR markers on tests that
make no HTTP calls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test against both latest (0.14.21) and 0.13.0 (first version with
  llama_index_instrumentation as a separate package)
- Set min_version="0.13.0" on LlamaIndexIntegration
- Install llms/embeddings packages unpinned in nox since they are
  tightly version-coupled to llama-index-core

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When both LlamaIndex and OpenAI integrations are active, skip token
metrics on the LlamaIndex LLM span so only the OpenAI leaf span owns
token accounting. Same pattern as CrewAI's _litellm_owns_leaf_span().

Also fix CI: add pylint disable=import-error for optional imports,
run ruff format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add LlamaIndex packages to the lint environment instead of suppressing import errors, tighten dispatcher state checks, and improve optional dependency fallback behavior.

Avoid double-counting async OpenAI token metrics, preserve block-backed message content, improve error detail logging, and add VCR coverage for LLM error handling.
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) merged commit 11c2016 into main Apr 27, 2026
82 checks passed
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) deleted the feat/llamaindex-integration branch April 27, 2026 22:56
Abhijeet Prasad (AbhiPrasad) added a commit that referenced this pull request Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bot] Add LlamaIndex native SDK integration for query engine, agent, and workflow instrumentation

1 participant